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JOINT RANGE OF RENYI ENTROPIES 
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The exact range of the joined values of several Renyi entropies is determined. The 
method is based on topology with special emphasis on the orientation of the objects studied. 

Like in the case when only two orders of Ronyi entropies arc studied one can parametrize 
upper and lower bounds but an explicit formula for a tight upper or lower bound cannot 
be given. 
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1. INTRODUCTION 

Let P = {pi,P2, ■■■,Pn) be a probability vectors. For a e K\{0, 1} the Renyi entropy 
of P of order a is defined as a number in [0; 00] given by the equation 

HAP) = ^log (y.P? 

This definition is extended by continuity so that 

-logminpi ; 

i 

log(number of p, ^0); 
-^Pilogpi ; 

i 

— log max Pi . 

i 

The Renyi entropy Hq is essentially the Hartley entropy, and was one among other 
sources of inspiration to Shannon's information theory. The Renyi entropy of order 
00 is also called the min-cntropy and essentially related to the " probability of error" . 
The Renyi entropy H2 is related to index of coincidence and other quantities used 
for special purposes in crypto analysis, physics etc. [2,8]. 

For all a the Renyi entropy Ha has the nice property of being additive on product 
measures. In noiseless source coding for finite systems one wants to avoid very long 
code words. For such systems the Renyi entropy of some order a < 1 (depending 



H-00 (P) = 
HoiP) = 
Hi{P) = 

Hoo (P) = 
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on the memory of the system) determines how much the source can be compressed. 
Renyi entropies are also related to general cut-ofF rates and " guess-work moments" 
[1-4]. 

The relation between Hq and Hi is given by the simple inequality 

Hi (P) < Ho (P) . 
This is a special case of the general result that 

a -> ffa (P) 

is a strictly decreasing function except for uniform distributions where it is constant, 
which follows from a simple application of Jensen's Inequality. The relation between 
Hi and Hoc has been determined independently in various articles [3,5,6,11,12]. 
The relation between Shannon entropy and H2 has been studied in [7] and in more 
detail in [8]. The result is illustrated on Figure [T| and by the following theorem. 

Theorem 1 The the upper bound on H2 (P) given Hi (P) is attained by a mixture 
of uniform distributions on k and k+l points where k is determined by the condition 
log fc < Hi (P) < log (fc + 1) . The lower bound on H2 (P) is attained by a mixture 
of the uniform distribution on n points and a uniform distribution on a singleton. 




1 bit 



Figure 1: Range of P ^ {Hi (P) , H2 (P)) for a four element set. 

In this paper we shall generalize this result and determine the joint range of 
several Renyi entropies. In general the boundary can be parametrized, but upper 
and lower bounds cannot be given by explicit formulas. The reason is that the 
inverse of the function s — *■ H^ {sUk + (1 — s) Uk+i) , where Uk and Uk+i are uniform 
distributions, is in general not an elementary function. 
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Recently the joint range of Renyi entropies has been used to determine the relative 
Bahadur efficiency of various power divergence statistics [9, 10]. In these papers the 
joint range of Hi and was used with a reference to [8] where the general result 
for comparison of two Renyi entropies was mentioned without proof. In some cases 
in physics, joint values of H2 (P) and (P) can be measured or computed and one 
is interested in bounds on Hi [13]. In order to get bounds on Hi one is interested 
in the exact range of the mapping 

^:P^{Hi{P),H2{P),H3{P)). 

In this paper the methods developed in [8] will be refined in order to be able to 
describe the joint range of in principle any number of Renyi entropies of positive 
order. We restrict our attention to non-negative orders because these are the most 
important for applications and because Renyi entropies of negative orders are not 
continuous near uniform distributions. Although the method is very general we shall 
only go into details in the cases where two or three Renyi entropies are compared. 
The main result is that the range has a boundary that can be parametrized by 
certain mixtures of uniform distributions. 



2. REDUCTION TO MIXTURES OF UNIFORM DISTRIBUTIONS 

A probability vector P on a set with n elements can be parametrized by its point 
probabilities as {pi,P2, ■■■,Pn) where pj > and 



E 

i=i 



Pj 



Here we shall assume that n is fixed so that that Hq (P) < log n. In order to study 
the range of P (iJc«i {P) , (P) , • • • , i?a„ {P)) we first consider the related 
map 

T^iog(Epr) 



v 



The matrix of partial derivatives is 



l-ar 



log (E^r) 

Hp 3 



(1) 



Oil 



<»2-l 

l-a2 



Pi 



\ — OCr 



-1 



ai P: 

Ep^ 

02-1 

Q2 P2 

i-«2 Epr 



am P2 

l-Cm EpT 



Oil 

Q2-1 

1-02 EpJ^ 



Ep"' 
1-0:2 EpJ^ 



l-Om Ep""* 1-Om EpJ' 
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If this matrix has rank m + 1 in a neighborhood of a point P —{pi,P2, •■•,Pn) then 
the map ([T]) is open, i.e. it maps open sets into open sets and a neighborhood of P 
is mapped into a neighborhood of the image. 

Next we show that if P has m + 1 different point probabihties then P is mapped 
into an interior point in the range. Therefore, assume that P has m + 1 different 
point probabihties. For simphcity we may assume that these m+ 1 different point 
probabihties are the first ones and that < pi < P2 < ■ ■ • < Pm+i- Then 



ai Pi 

1-02 Ep? 



Pi 



Ep? 



ai P2 

1-ai J^p" 

Q2 P2 

l-aa Ep° 



P2 



ai Pj, 



Ej 



Ht^ n 



^4=1 



=1 



pT-' 



Pi 



EpJ' 

a2 Pm^~^ 

1-Q2 EP°^ 



1 

pT-' 



02 Pto + 1 

i-a2 ep7' 



P2 



Ep° 

1 

ni 

,Q2 — 1 



(2) 



1 



Pm+1 

„Q2-1 



1 



)with 



Note that the last row can be written as ( P2~ ' ' ' Pn 

a = 1. The last determinant is a generalization of the Vandermonde determinant. 
Like a Vandermonde determinant, it is non-zero if and only if the entries are different, 
which is the next we have to prove. 



Lemma 2 Assume that < xi < X2 < 
generalized Vandermonde determinant 



< Xi and Pi < P2 < ■ ■ ■ < Pe- Then the 



det 



is non-negative. It is zero if and only if there exists j £ {1,2, ■ ■ ■ £ — 1} such that 



Proof The proof is by induction in i. For £ = 1 the generalized Vandermonde 
determinant is obviously positive. Assume that the result holds for ^ = fc — 1. We 
have to prove it for £ — k. First we have 



n 



J2-I3i 



X 



1 



X 



1 

A- Pi 
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Therefore without loss of generahty we may assume that (3i 
to prove that 

' 1 1 ••• 1 1 



0. Therefore we have 



1 



^k-1 



is non-negative. If Xk — x^^i the last two columns are identical and determinant 
is zero so it is sufficient to prove that the partial derivative with respect to Xk is 
non-negative. The partial derivative is 



1 



^Pk 



1 



1 







Similarly we may take partial derivatives with respect to Xk^i, Xk-2, • 
and get 



, X3 and X2 



1 







/33-1 





f32xt 



P2xt 

Psxt 



f3kx^' 





P2xt' 



P2XI' 

Psxl' 



(i2xl^ 



^2 



This is non-negative according to the induction hypothesis. I 

We see that if < ai < • • • < am < 1 then the determinant ^ is positive. It is 
easy to check that this is also the case with the relaxed condition < ai < • • • < Um- 
The Renyi entropies are symmetric in their entries. Therefore we may restrict 
our attention to probability vectors with increasing entries, i.e. < pi < p2 < • • • < 
Pm+i- The extreme points in the set of ordered probability vectors are the uniform 
distributions. Let Uk denote the uniform distribution (0, 0, • • • , 0, ^, • • • Let 
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ki,k2, ■ ■ ■ ,k£ be a sequence of difTerent numbers in {1, 2, • • • ,n}. Then the simplex 
formed by convex combinations of Uk-^ , 11^2, ■ ■ ■ , will shall be denoted A^^ j.^.- • ,ke 
and be given an orientation according to the sequence Uki , , • • • , Uki ■ Observe that 
if A;i > ^2 > ■ • ■ > km+i then the mapping /S.^^^^^,--- ,fem ~^ defined by 

/ T^log(Ep?) \ 

T^iog(Epr) 

has positive orientation if < ai < a2 < • • • < a^n- 



3. JOINT RANGE OF TWO RENYI ENTROPIES 

First we consider distributions on a set with n elements. We determine the joint 
range of and H^^ where we assume that < ai < a2- First we shall also assume 
that a\,a2 € ]0; oo[ \ {1} . Let $ denote the map 

Ho., (P) 

Ha2 {P) 

Assume that ki > k2 > k^. Then $ (Ukj) lies on the diagonal {{x,x) : x > 0} , 
and these points are ordered, 

where a = ai or a = a2. We know that Ha, (P) > H^^ {P) with equality if and 

only if P is a uniform distribution. Therefore $ restricted to Afej_fe2,fe3 must preserve 
orientation. We know that $ maps inner points of H^f^^^^^M into inner points of the 
range of $ so boundary points of the range of $ must have preimages that are 
boundary points of A^.^ fe^.fei- We follow the conventions from homology theory and 
calculate the boundary with orientation. The boundary of $ (A/c^^fe^^feg) is 

which is just another way of writing the closed curve from Uk-, to Uk-^ to Uk^, and 
back to (7fej . Therefore any point on the boundary of the range of $ must be the 
image of a mixture of two uniform distributions. 

Assume that fci > A;2 > fcs > ki. Then the simplices J^k^^k^M ''^'^'^ ^kiMM ^'^^ 
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both positively oriented and 



Ak2M ~ ^kiM + AkiM 

+ ^fe3,fc4 ~ ^fcl,fe4 + ^k3,ki 

^k2M - ^kiM + ^kiM 

= $ (Afei,fc, + Afc3,fe3 + Afe3,fc^ + Afe^,fcJ . 



We see that $ [AkiM) (^oes not contribute to the boundary of 

{Ak^,k2,k3 + ^kiMM) ■ 

Similarly ^(Afcj^fc^) does not contribute to the boundary. We may formulate this 
result as A^ does not contribute to the range if it is a diagonal in a quadruple. 
The non-diagonal simplices are A„^„_i, A„_i^„_2, • • • , A2,iand Ai_„. These form a 
closed curve 

A„_„_i + A„_i^„_2 + • • • + A2,i + Ai^„ 
and the boundary is the image of this curve, i.e. 

$ (A„,„_i + 

^n— l,n— 2 

H + A2,i + Ai^„) , 

This result easily extends to the cases where one or more of the orders equal 1 or 
oo. The upper bound does not depend on n so we get the following theorem. 

Theorem 3 Assume < ai < a2- Then the upper bound on H^^ (P) given H^-^ (P) 
is attained by a mixture of uniform distributions on k and k + 1 points where k is 
determined by the condition log k < H^^ (P) < log (fc + 1) . 

For distributions on set with n elements we also get a tight lower bound, but if 
we have no restriction on n the situation is a little more complicated. 

Theorem 4 Assume Q < a\ < a2- If P is a distribution on a set with n elements 

and (P) is fixed then a lower hound on is attained for a mixture of the 

uniform distributions Ui and C/„. If no restriction on n is given and if (P) > 
is fixed then a tight lower bound on H^^ (P) is given by 



Ha2 {P) > 



Ol2 



0, ifai < 1; 

-^-i^H^^iP), ifai>l. 



Proof If we have no restriction on n then the range is 

CSO 

0$(A„+i,„,i). 



n=2 
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So wc just have to determine the asymptotics of (f>(A„ i). The curve Ai „ has 
the parametrization Pt = ^, ■ ■ ■ , ^, ^ + 1 — , t € [0; 1] . Therefore the curve 
$ (A„^i) has the parametrization 



,^log((n-l)(A)- + (A + l-t)-) 



We have to study the asymptotics of this curve for n tending to infinity. There are 
several cases and they need separate analysis. 

Ccise ai > 1. We also have a2 > 1 so for a fixed value of t we get 

I i^jog{{n-i){ir+{i+i-tr) ) [ i^iog(i-i) 

for n tending to infinity. Hence the straight line with slope J^^i "^""^ is the 
boundary of the range. 

Case Q2 > 1 and ai < 1. First we assume that qi < 1. For a fixed value of the 
parameter t the Renyi entropy ifaa tends to a constant as above but Ha^ tends 
to infinity. For a fixed value of H^i {P) > the lower bound {P) > is 
tight. This bound is also tight for ai = 1 and can be obtained by letting ai 
tend to 1 from above or below. 

Case < a2 < 1- First assume that a2 < 1. If t = n^-^/"^ then 

/ T^log((n-l)(^)- + (^ + l-0"^) \ 
I T^log((n-l)(^)- + (^ + l-0"^) j 



_J_ log (n-^-^ + + 1 - ni-i/"^)"^) 

^ log + (n-V- + 1 - n^-V-)'^^) 

We see that the second coordinate tends to — log 2, while the first coordi- 
nate tends to oo. Therefore for a fixed value of (P) > the lower bound 
{P) > is tight. Tightness of this bound also holds for a2 = 1, which can 
be seen by letting a2 tend 1 from above or from below. ■ 

4. JOINT RANGE OF THREE RENYI ENTROPIES 

Determining the range of three Renyi entropies is done in the same way as in the 
previous section. We consider the map ^ given by 

Ha, (P) 

Ha, (P) 
Ha, (P) 

First we consider the situation where the domain consist of distributions on n points. 
Boundary points of must be images of mixtures of three uniform distributions. If 
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n>m>£>k>l then the restriction of to the simphces An^m,e,k or to A^^j i 
conserves orientation. Therefore 

a* {An,m,e,k + Am,e,k,i) = {dAn,rn,e,k + dAm,e,k,i) 

_ ( Am,e,k — An,e,k + An,m,k ~ An^rn,e \ 

\ +Ae^k,l — A^^k^i + ArnJ^l — Arn^e^k J 

— ( An^rn,k — An^m,e \ 

\ +Ai^k,l — Am,k,l + AmJ,l J 

We see that A^^^^^ gives no contribution to the boundary and therefore only sim- 
plices Am,e,k with either m = n or fc = 1 give a contributions to the boundary. 

lin>m>£>k>l then the restriction of ^ to the simphces An,m,k,i or to 
Am,e,k,i conserves orientation. Therefore 

a* {An,m,k,l + Am,e,k,l) = 9* {dAn,m,k,l + dAm,i,k,l) 

^■qj { Ajn,k,l Aji^k,! ~t~ An^ni.l 



+Ae^k,l — Am,k,l + Am,e,i — Am,e,k 

V +Ai,k,i + Am,e,i - Am,e,k J ' 



Wc sec that the simplex A„i,k,i gives no contribution to the boundary of the range 
of \E'. So if m < n the simplex Am,k,i can only give a contribution to the boundary 
if there exist no natural number i such that m > i > k, i.e. k = m — 1. In the 
same way we can show that a simplex of the form A„ „, £ will only contribute to the 
boundary if ^ = m — 1 and that a simplex A^^^^i only contributes if m = n — 1 or if 
m = 2. Thus the boundary of the range consist of images of the simphces Ajn,m-i,i 
and of the form An,m,m-i- Here we notice that 

/ n n—1 \ n n—1 



d I A^jm— 1,1 An^rn,m.—1 ] — ^A^^^— 1,1 9An^m,m—l 

Vm=3 m— 2 / m— 3 m=2 

n 

= ^3 — Am,l + Am,m-l) 

n-1 

~ {Am,m—1 ~ An^m—1 + An^m) — 0, 



m=3 



^=2 



SO that 

n n- 

Aui^m—1,1 A 



n,m,m— 1 

m=3 m=2 



is a closed surface and that the range of ^ has the image of this surface as boundary. 
It is possible to describe the situation in more detail. Let $ denote the map 

Ha, (P) 
Ha, (P) 
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Figure 2: The left diagram illustrates the range of A4.2,i and A4.;j.2 with orientation 
(n = 4). The range \E' appHed to these simpUces give lower bounds on Hs- The right 
diagram illustrates the range of A3_i^2 and A4^i_3 with orientation (n = 4). The 
range applied to these simplices give upper bounds on Hs. 



n 

Then $ restricted to A^.m-i,! is a homeomorphism. If 

m— 3 

then there exist a unique m and unique weights x,y,z > that sum up to 1 such 



that P = X ■ Um + y ■ Um-i + z ■ Ui. For any distribution Q with $ (Q) = 




we have {Q) < H^^ (P) . Thus, A„,.,„_i_i gives tight upper bounds on 

rn='A 

in terms of and H^^- We notice that this upper bound does not depend on 
n. Similarly, the lower bound on Ha^ for fixed and is determined by the 

ri-l 

surface A„_m,m-i and just as in the case of two Renyi entropies the lower bound 

m=2 

will depend on n. 
5. DISCUSSION 

The result can be seen as a generalization of the result in [8]. The essential step in the 

whole construction is the positivity of the generahzed Vandermonde determinant. 
Therefore the construction can be iterated so that one in principle can determine 
the boundary of the range of any number of Renyi entropies of positive order. 
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